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1.  INTRODUCTION 


1.1  SIGNIFICANCE  OF  THE  RESEARCH 


This  paper  models  Unbalanced  flows  due  to  Asynchronously 
spawned  Parallel  tasks  (UAP)  in  a  distributed  system  using 
generalized  queueing  network  models  as  developed  by  Jackson 
(1963),  Gordon  and  Newell  (1967),  Baskett  et  al  (1975),  Reiser 
and  Kobayashi  (r975),  and  Denning  and  Buzen  (1978).  The  acronym 


UAP 

will 

be  used 

throughout  the  paper  to 

refer  to 

unbalanced 

flows 

(i 

.e.  number 

of  transactions  leaving 

a  server 

is  not  the 

same 

as 

number  of 

transactions  entering 

that  server)  due  to 

asynchronously  spawned  parallel  tasks  which  are  assumed  to  run 
independently  of  each  other  except  for  resource  contention. 

The  significance  of  this  research  is  as  follows:  (1)  A 
decomposition  method  is  applied  to  incorporate  the  workload  due 
to  UAP  —  a  primary  effect  on  speed  performance  —  into  queueing 
network  models;  (2)  Formulae  for  open  queueing  networks  with  UAP 
are  developed;  (3)  An  algorithm  based  on  Buzen' s  convolution 
algorithm  is  developed  to  test  the  necessary  and  sufficient 
condition  for  closed  system  stability;  and  (4)  Formulae  for 
closed  queueing  networks  with  UAP  are  developed  based  on  the 
algorithm.  As  a  result,  performance  measures  are  assessed  more 
accurately.  A  cost  effective  software  tool,  using  this 
methodology,  has  been  developed  (Wang  and  Madnick,  1984)  to 
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analyze  system  architectures  with  UAP  so  that  performance 
implications  can  be  identified  early  in  the  design  process. 

The  discussion  in  this  paper  uses  as  an  example  the  INFOPLEZ 
data  Storage  hierarchy  (INFOPLEZ  is  a  database  computer  research 
project  at  the  MIT  Center  for  Information  Systems  Research;  the 
theory  of  hierarchical  decomposition  is  applied  in  this  research 
to  structure  hundreds  of  microprocessors  together  to  realize  a 
low  cost  data  storage  hierarchy  with  very  large  capacity  and 
small  access  time.  See  Madnick,  1973,  1977,  1979,  Lam  and 
Madnick,  1979,  and  Wang  and  Madnick,  1981,  1984).  However,  the 
methodology  employed  in  this  research  is  generalizable  to  other 
distributed  information  systems  (Trivedi  and  Sigmon,  1981,  Geist 
and  Trivedi,  1982,  and  Goyal  and  Agerwala,  1984). 

1.2  PURPOSE  AND  BACKGROUND  OF  THE  RESEARCH 

The  goal  of  system  development  is  to  produce  systems  that 
satisfy  their  specifications  when  completed  while  minimizing 
costs  and  time  required.  A  key  to  minimizing  development  cost 
and  time  is  to  determine  whether  the  system  will  meet  its 
functional  and  performance  requirements  as  early  as  possible  in 
the  development  process.  This  will  avoid  wasted  work  toward  an 
unsatisfactory  implementation  and  the  subsequent  rework.  To  this 
end,  a  cost  effective  tool  to  evaluate  system  performance  is 
essential  (Gagliardi,  1982). 

The  use  of  analytic  performance  models,  instead  of 
simulation  models,  has  become  increasingly  popular  recently 
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(Graham,  1978)  because  the  analytic  approach  is  more  cost 
effective  than  simulation.  Figure  1-1  illustrates  the  difference 
in  terms  of  CPU  time  and  dollar  cost  between  the  analytic  model 
and  the  simulation  model  that  the  authors  have  conducted  for  a  5 
processor  4  level  INFOPLEX  design,  called  P5L4.  The  cost  of 
running  the  analytic  model  was  found  to  be  from  200  to  2000  times 
less  expensive  (Wang  and  Madnick,  1984).  An  even  more  important 
practical  difference  was  that  a  single  simulation  run,  to  test 
one  design  of  the  fairly  simple  P5L4  system,  costs  about  $100  — 
whereas,  the  comparable  analytic  calcuation  costs  around  5  cents. 
At  $100  or  more  per  design,  it  may  be  difficult  for  a  researcher 
to  justify  the  costs  of  exploring  the  hundreds  or  thousands  of 
design  and  parameter  alternatives.  At  5  cents  per  design  studied, 
using  the  analytic  model,  the  researcher  can  freely  explore  many 
diverse  alternatives  and  a  wide  range  of  parameters  in  search  for 
an  optimal  design. 

In  order  to  obtain  accurate  results,  all  significant  factors 
effecting  performance  should  be  captured  in  the  analytic  model. 
UAP  has  been  found  to  have  a  primary  effect  on  performance  (Wang 
and  Madnick,  1981,  1984).  Unfortunately,  networks  with  UAP  did 
not  have  an  analytically  tractable  solution  because  the  input 
flow  and  the  output  flow  are  not  balanced  at  the  places  where 
parallel  tasks  are  spawned,  a  violation  of  the  principle  of  job 
flow  balance  (Denning  and  Buzen,  1978:  the  principle  of  job  flow 
balance  says  that  the  number  of  customers  that  flow  into  a 
service  facility  equals  to  the  number  of  customers  that  flow  out 
of  the  facility  when  the  system  is  in  the  steady-state). 
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RUN 

Simulation 

PERIOD  CPU-TIME  COST$ 

Analytic 

CPU-TIME  COST$ 

1 

10 

ms 

434 

97.33 

12 

0. 

05 

2 

3 

ms 

270 

61.70 

12 

0. 

05 

3 

2 

ms 

349 

78.22 

12 

0. 

05 

4 

2 

ms 

308 

70.32 

12 

0. 

05 

5 

1 

ms 

205 

47.77 

12 

0. 

05 

6 

1 

ms 

351 

79.02 

12 

0. 

05 

7 

.5 

ms 

453 

101.06 

12 

0. 

05 

8 

.3 

ms 

290 

65.55 

12 

0. 

05 

9 

.05 

ms 

47 

13.09 

12 

0. 

05 

10 

.05 

ms 

38 

10.54 

12 

0. 

05 

Simulation  CPU-TIME  is  in  CPU  seconds  on  an  IBM  370/168. 
Analytic  CPU-TIME  is  12  CPU  seconds  per  run  on  a  PRIME/850. 
"Cost$"  is  in  dollars  for  the  overall  charge  per  run. 

"ms"  in  the  table  means  milli-seconds . 

To  attain  steady-state,  simulation  periods  of  10  ms,  or  more, 
are  usually  needed. 

Figure  1-1;  A  Comparison  of  costs:  Simulation  vs.  Analytic. 


A  simplified  1  processor  2  level  INFOPLEX  model,  called 
P1L2,  is  given  below  to  illustrate  the  UAP  phenomenon.  Consider 
the  routing  diagram  (Figure  1-2)  of  the  P1L2  model  which 
processes  read  and  write  operations.  Suppose  80%  of  the  customers 
request  the  read  operation  (RP1)  and  20%  request  the  write 
operation  (WP1);  and  the  read  operation  has  100%  locality,  i.e. 
the  requested  data  is  always  found  at  the  level  one  device,  D1 . 
The  read  operation  is  serviced  by  the  level  one  processor,  PI, 
first,  then  the  data  is  retrieved  from  D1  and  returned  to  the 
reference  source  (SINKM).  The  write  operation  is  acknowledged 
immediately  by  PI  to  the  reference  source  (SINKM); 
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in  parallel,  the  data  is  updated  at  Dl,  stored-behind  to  the 
level  2  device,  D2 ,  then  the  asynchronously  spawned  task 
terminates  (SINKU). 

Note  that  class  WP1  leaves  facility  PI  with  "a  routing 
probability  one"  to  SINKM  and  "a  routing  probability  one"  to  WD1 
as  indicated  by  the  dash  line;  i.e.  the  out-flow  is  twice  as  much 
as  the  in-flow,  violating  the  principle  of  flow  balance. 

1.3  LITERATURE  REVIEW 

Several  studies  have  been  attempted  to  generalize  queueing 
network  models  to  include  parallel  processing.  Browne,  Chandy, 
Horgarth,  and  Lee  (1973)  investigated  the  effect  on  throughput  of 
multiprocessing  in  a  multiprogramming  environment  using  the 
central  server  model  approach.  Reiser  and  Chandy  (1979)  studied 
the  impact  of  distributions  and  disciplines  on  multiple  processor 
systems.  Towsley,  Chandy,  and  Browne  (1978)  developed 
approximate  queueing  models  for  internal  parallel  processing  by 
individual  programs  in  a  mult iprogrammed  system  based  on  the 
central  model  approach  and  "Norton  theorem".  Price  (1975) 
analyzed  models  of  multiple  I/O  buffering  schemes.  Maekawa 
(1976)  and  Peterson  (1979)  modeled  a  number  of  CPUrlO  overlap 
cases.  These  studies,  although  valuable,  do  not  fit  systems  which 
have  a  generalized  topology  and  the  UAP  phenomenon. 

Modeling  the  UAP  phenomenon  for  generalized  queueing  network 
systems  is  a  relatively  new  topic,  first  reported,  to  our 
knowledge,  by  Heidelberger  and  Trivedi  in  1982.  In  that  work,  an 
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Proof:  Define  (Reiser  and  Kobayashi,  1975)  the  p.g.f.  for 
P(n,(M),  n, (U) ,  ...,  nM (M) ,  n*(U))  as 
G(Z  ,9)  *  n  *<(  /MU)  *  z  i  ( U )  +  p  i  (M)  *  2  i  (M)  *  d) 
where  z,  is  the  p.g.f.  transformation  variable  for  facility  i;  9 
is  a  factor  associated  with  the  main  chain  to  insure  that  main 
chain  population  is  fixed  to  N;  the  products,  is  taken  from  1  up 
to  M,  and  ♦,(£)  =  1/(1-:)  for  FCFS,PS,  and  LCFSPR.  The  p.g.f. 
is  found  as  the  coefficient  of  in  a  power  series  expansion  of 
G(Z  ,9)  in  9,  denote  it  G*(Z).  It  follows  that 

G*  (  Z )  =  C  *  atf(N)  *  n  4* ,  {pi  ( U)  *  2 ,  ( U )  +  /MM)  *  Z  i  (M)  *  9) 

To  obtain  the  p.g.f.  of  the  marginal  distribution  of  the 
closed  main  chain,  let  2,(U)=1.  It  follows  that 

G*  (2( (U) =1 ) 

=  C  *  ^(N)  *  n  ♦  (/MU)  +  />  i  (M)  *  2,(M)  *  9) 

*  C  *  e^(N)  *  n  l/(  1  -  Pi(U)  -  yo  <  (M)  *  2  i  (M)  *  9  ■) 

=  c  *  (n  1/(1  -  p,( U) ) )  *  a*(N) 

*  n  l/(l-(  p,{ M)  *  Z,(M)  *  9  /  (  1  -  /MU)))) 

=  ( 1/G(N)  )  *  (  zn  (/MM)  *  2  i  ( M )  /(  1  -  />1(U)))ni(MM 
where  the  summation  is  taken  over  all  possible  states  of  S(N,M)  = 
(  (n, (M) ,  n M ( M ) ) |  n, (M)  +  ...  +  n M ( M )  =  N,  and  n, (M)  >  0  for 

i 

all  i  }.  But  this  is  exactly  the  p.g.f.  for  CPFSCQN  (Lavenberg, 
1983)  with  the  traffic  intensity  inflated  by  (1-/MU))'1  f°r 
facility  i.  Q.E.D. 

From  the  marginal  distribution  above,  it  is  not  difficult  to 
show  (Heidelberger ,  1982,  Mayrhauser,  1983)  that  f  is  CMD , 
assuming  that  there  exists  at  least  a  pair  of  (D,(M),  Di(U))  such 
that  D , (M) >0  and  Dj(U)>0.  With  the  corollary  and  the  CMD 
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3.4  CLOSED  QUEUEING  NETWORKS  WITH  UAP 

For  closed  queueing  networks  with  UAP,  a  mixed  network  with 
the  closed  main  chain  and  the  open  UAP  chain,  as  illustrated  in 
Figure  3-1,  can  be  obtained  following  the  discussion  in  section 
3.1.  Since  Xe(U)  *  X0(M)  *  V(U)  where  X0(M)  is  evaluated  through 
a  nonlinear  function  of  X0(U)  (Reiser  and  Kobayashi ,  1975).  It 
follows  that  X0(U)  =  f(X0(U))  *  V(U)  where  f  is  a  nonlinear 

function.  To  solve  the  nonlinear  equation,  a  couple  of  issues 
have  to  be  addressed  first: 

A)  What  are  the  properties  of  f? 

B)  what  is  the  necessary  and  sufficient  condition  for  the 
network  to  be  stable? 

A  corollary  based  on  Reiser  and  Kobayashi 's  theorem  (1975) 
on  PFMQN  is  shown  below  to  settle  issue  A;  and  a  lemma  is  proven 
to  settle  issue  B  which  leads  to  an  iterative  procedure  for  the 
closed  network.  The  IS  discipline  is  excluded  from  this 
subsection.  Its  difference  from  other  disciplines  will  be 
discussed  at  the  end  of  the  section. 

A)  Corollary:  An  equivalent  closed  network(EN)  of  the  main  chain 
for  the  mixed  network(MN),  as  illustrated  in  Figure  3-2,  can  be 
obtained  by  inflating  the  main  chain  traffic  intensities,  i.e.  by 
replacing  /^(M)  by  /^(M)  /  <  1— /» i  (U) )  for  i  =  1,  ...,  M. 
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It  can  be  shown  (Sauer  1981,  Lazowska  1984)  that  throughput, 
utilization,  mean  queue  length,  and  response  time  are  computed  as 
shown  in  Table  3-1.  Note  that:  1).  U^M)  would  be  the  sum  of 
the  products  of  throughputs  and  mean  service  times  of  all  the 
classes  if  there  were  multiple  classes  of  customers  at  facility  i 
for  the  main  chain;  the  same  mechanism  applies  to  the  UAP  chain. 
2).  The  denominator  of  Nj(M)  is  U,  which  quantifies  the  resource 
contention  between  the  UAP  chain  and  the  main  chain.  3).  The 
"system  response  time"  in  a  flow  unbalanced  network  is  defined  as 
the  time  to  complete  the  main  chain,  R0(M),  since  that  is  the 
only  observable  completion  seen  by  the  external  world. 


Facility  i 

FCFS , PS , LCFSPR  discipline 

Xi  (M) 

X0  (M)  *  V ,  (M) 

Xi  (U) 

xc(U)  *  V, (U)  /  V(U) 

Xi 

X *  (M)  +  X ^  (U) 

U1  (M) 

X i (M)  *  S i (M) 

Ui  (U) 

X,(U)  *  S  i  (U) 

Ui 

U i (M)  +  U i (U) 

N  i  (M) 

U i (M)  /  (1-U() 

N i  (U) 

U i (U)  /  ( 1~U i ) 

Ni 

N,(M)  +  N , ( U ) 

N, (M)  /  X i (M) 

N, (U)  /  X, (U) 

N,  /  X, 

Ro(M) 

R 1 (M)  +  ...  +  Rc (M) 

Ro 

R i  +  ...  +  Rc 

Table  3-1:  Formulae  for  Open  Queueing  Networks  with  UAP. 
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chain  with  its  workload  contributed  from  both  the  main  chain  and 
the  UAP  chain;  section  3.3  discusses  the  formulation  of  useful 
performance  measures  for  open  queueing  networks  with  UAP.  On  the 
other  hand,  if  the  original  network  is  a  closed  network,  then  we 
have  a  mixed  network  with  the  closed  main  chain  and  the  open  UAP 
chain.  Section  3.4  discusses  the  necessary  and  sufficient 
condition  for  the  closed  network  to  be  stable  and  an  iterative 
procedure  which  computes  the  system  throughput. 

It  is  extricable  now  to  formulate  networks  with  UAP.  Let 
the  summation  of  visit  ratios  over  all  the  cuts  in  section  3. 2. A, 
V(U),  denote  the  "internally  generated"  visit  rate  of  the  UAP 
chain.  Note  that  "(M)"  will  denote  an  open  chain  in  section  3.3 
and  a  closed  chain  in  section  3.4. 

3.3  OPEN  QUEUEING  NETWORKS  WITH  UAP 

For  an  open  queueing  network  with  UAP,  the  network  arrival 
process  is  assumed  to  be  poisson  with  a  constant  rate  XD.  By 
solving  the  extended  routing  matrix  introduced  in  section  2,  one 
can  obtain  the  visit  ratios  for  all  classes,  hence  V(U).  Since  xQ 
is  given,  XQ(U)  is  also  determined,  specifically,  X0(M)  =  xc  and 
X0(U)  =  x0  *  V(U).  For  instance  suppose  xc  =  5  customers/sec  in 
Figure  1-2,  then  the  UAP  chain  (SOURCEU,  WD1 ,  WD2 ,  SINKU)  ,as 
shown  in  Figure  1-4,  has  an  arrival  rate  of  1  customer/sec. 

Since  the  network  can  be  aggregated  to  an  open  single  chain 
network,  its  stability  follows  from  OPFSCQN,  i.e.  the  network  is 
stable  if  and  only  if  U^  <1  for  all  facilities  in  the  network. 
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balanced  main  chain  with  its  classes  in  the  set  {Rc}  and  many 
open  chains  with  their  classes  in  the  set  (R)  -  {Rc} .  Therefore, 
the  classes  in  the  main  chain  and  the  classes  in  the  open  chains 
are  disjoint. 

However,  it  has  been  pointed  out,  in  section  2,  that  a  split 
may  split  again,  so  the  open  chains  may  themselves  be  flow 
unbalanced.  To  solve  the  problem,  it  is  logical  to  cut  all  the 
additional  unbalanced  flows  in  the  open  chains  continuously  (and 
insert  "internal  sources"  which  generate  equivalent  flow  rates  as 
those  of  the  open  chains  before  the  cuts)  until  all  flows  are 
balanced,  forming  additional  open  chains. 

It  is  assumed  that  service  time  distributions  and  service 
disciplines  of  the  facilities  in  the  network  follow  those  of 
classical  product  form  queueing  networks  (Baskett  et  al);  in 
addition,  the  unbalanced  flows  which  run  independently  of  one 
another  are  assumed  to  arrive  at  their  destinations  as 
independent  poisson  processes  (this  assumption  is  also  adopted  by 
Goldberg  (1983)  and  Heidelberger  (1983).  However,  as  Burke  (1972) 
pointed  out,  these  processes  are  not  Poisson  in  general.  The 
simulation  studies  that  the  authors  have  conducted  indicate  that 
this  is  a  fairly  robust  approximation.  The  validation  reported 
by  Goldberg,  et  al  (1983)  provides  further  support  for  this 
assumption.  It  follows  that  the  OPFMCQN  result  can  be  applied  to 
aggregate  the  additional  open  chains  discussed  in  the  last 
paragraph  to  a  single  open  chain  —  the  UAP  chain. 

If  the  original  network  is  an  open  network,  then  the  OPFMCQN 
result  can  be  applied  again  to  make  the  overall  network  a  single 
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main  chain;  and  D,(M)  ■  S<(M)  *  (M)  is  the  product  of  visit 
ratio  and  mean  service  time  of  facility  i  for  the  main  chain. 

3.2  QUEUEING  NETWORKS  WITH  UAP 

It  was  noted,  in  section  2,  that  a)  UAP  can  occur  in  many 
classes  within  a  queueing  network;  b)  an  input  to  a  class  that 
causes  UAP  may  be  the  output  from  another  UAP  class;  and  c)  all 
the  additional  unbalanced  flows  are  defined  to  belong  to  the  UAP 
chain  —  a  single  chain.  It  is  natural  to  ask  whether  the  flows 
of  the  transformed  network  would  be  balanced,  and  what  kind  of 
relationship  would  exist  between  the  main  chain  and  the  UAP 
chain.  These  questions  are  answered  below: 

If  one  cuts  the  additional  b-1  unbalanced  flows  from  a  class 
which  is  UAP  with  degree  b  and  inserts  "internal  sources" 
( SOURCEU )  which  generate  customers  with  equivalent  flow  rates  as 
those  of  the  network  before  the  cut,  then  following  the 
assumption  that  unbalanced  flows  run  independently  of  one  another 
except  for  resource  contention,  the  b-1  unbalanced  flows  will 
form  b-1  new  open  chains  which  will  not  interact  with  the  main 
chain.  If  all  the  additional  unbalanced  flows  (spawned  from  the 
classes  which  are  UAP  and  connected  to  the  main  chain)  are  cut 
from  the  main  chain,  then  the  flow  in  the  main  chain  will  be 
balanced,  as  illustrated  in  Figure  1-3. 

Let  (R)  denote  the  set  of  classes  in  the  network  before  the 
cuts  and  {Rcl  denote  the  set  of  classes  in  the  main  chain,  as 
illustrated  in  Figure  2-1  and  2-2.  It  follows  that  we  have  the 
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C  total  number  of  classes  in  the  network. 

CMD  continuous  and  monotonically  decreasing 

D  V*S;  the  product  of  visit  ratio  and  mean  service  time. 

FCFS  first  come  first  serve. 

f  x0(M)*f (Xe(U) ) ;  the  main  chain  throughput  as  a  nonlinear 

function  of  the  UAP  chain  throughput. 

IS  infinite  server. 

LCFSPR  last  come  first  serve  preemptive  resumable. 

M  number  of  service  facilities  in  the  network. 

N  mean  number  of  customers (mean  queue  length  including  the 

one  in  service), 
n  number  of  customers. 

PS  processor  sharing. 

p.f.s.  product  form  solution. 

p.g.f.  probability  generating  function. 

R  mean  response  time. 

S  mean  service  time. 

U  utilization. 

UAP  unbalanced  flows  due  to  asynchronously  spawned  parallel 
tasks. 

V  visit  ratio. 

X  throughput. 

X  arrival  rate. 

P  traffic  intensity. 

Example:  S,(M)  means  the  mean  service  time  of  facility  i  for  the 
main  chain;  V, (M)  means  the  visit  ratio  to  facility  i  due  to  the 
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3 .  ANALYTI C  FORMULATION 


To  present  the  paper  concisely,  service  rate  is  assumed  to 
be  fixed,  i.e.  load  independent  (Denning  and  Buzen,  1975, 
Lavenberg,  1983  or  Lazowska,  1984).  The  following  useful  results 
and  models  -  (Kleinrock,  1975,  1976,  or  Lavenberg,  1983)  are 
recapitulated  in  the  Appendix  to  serve  as  a  point  of  departure 
for  the  discussion  in  this  section:  1)  Little's  formula,  2) 
non-product-form  queueing  networks,  3)  product  form  queueing 
networks,  4)  single  chain  queueing  networks,  5)  Open  product  form 
single  chain  queueing  networks  (OPFSCQN) ,  6)  Open  product  form 
multiple  chain  queueing  networks  ( OPFMCQN ) ,  7)  closed  product 
form  single  chain  queueing  networks  (CPFSCQN) ,  8)  Convolution 
algorithm,  and  9)  product  form  mixed  queueing  networks  ( PFMQN ) . 


3.1 

NOTATIONS 

Notations 

used  in  this  paper  are 

listed  below: 

subscripts: 

i 

denotes 

an  individual  service 

facility. 

o 

denotes 

the  overall  network. 

(M) 

denotes 

the  main  chain. 

(U) 

denotes 

the  UAP  chain. 

O1 

denotes 

the  ith  iteration. 

Sl_ 

notations: 

B  bottleneck  facility  (therefore  chain)  throughput. 


PAGE  11 


SOURCEM 

RP1 

RD1 

WP1 

WD1 

WD2 


RP1  WP1  RD1  SINKM  WD1  WD2  SINKU 

'  .8  .2  0  0  0  0  O' 

0  0  1  0  0  0  0 


Figure  2-1;  The  Extended  Routing  Matrix  for  Figure  1-2 


.8  .2 

0  0 


The  visit  ratios  of  the  classes  in  Rc  can  be  obtained  from 
the  visit  ratio  equations  (5,  p.237),  viz., 


IV,  *  pftj 


j  *  1,  • ♦ • »  Ci 


Alternatively,  the  visit  ratio  equations  can  be  applied 
directly  to  the  extended  routing  matrix  R  to  obtain  all  the  visit 
ratios  of  the  classes  in  R. 
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with  a  part  of  the  main  chain.  Therefore,  a  unique  syntactic 
definition  exists  for  each  UAP  network. 

Classical  queueing  network  models  cannot  be  applied  to 
analyze  UAP  directly  because  of  the  unbalanced  flows  mentioned. 
An  extended  routing  matrix  is  introduced  below  to  accommodate  the 
problem.  Let  R  denote  the  extended  routing  matrix  of  an  UAP 
network  where  a  row-sum  may  be  greater  than  one.  The  extended 
routing  matrix  R  for  Figure  1-2  is  shown  in  Figure  2-1.  Let  Rc 
denote  the  unextended  routing  matrix  which  excludes  the  UAP  chain 
of  the  network.  The  unextended  routing  matrix  Rc  which  excludes 
the  UAP  chain  (SOURCEU,  WD1,  WD2,  SINRU)  is  shown  in  Figure  2-2. 
Elements  in  R  and  Rc  are  the  routing  probabilities  p1tJ's. 

Define  the  visit  ratio  of  a  class,  Vc,  as  the  mean  number  of 
requests  of  the  class  to  a  service  facility  per  customer.  Define 
the  sum  of  visit  ratios  of  all  exogenous  sources,  V0,  in  an  open 
system  to  be  one.  In  a  closed  system,  the  outputs  feedback  to 
the  system  inputs;  the  sum  of  visit  ratios  of  the  system  inputs 
is  also  defined  to  be  one. 
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For  example,  in  Figure  1-2,  UAP(WP1,P1)  ■  2.  Note  that  (a)  UAP 
can  occur  in  many  classes  within  a  queueing  network,  for  instance 
acknowledgements  may  take  place  at  different  levels  of  a  data 
storage  hierarchy;  and  (b)  the  inputs  to  a  class  that  cause  UAP 
can  be  the  outputs  from  other  UAP  classes.  For  instance,  a  split 
from  an  acknowledgement  may  split  again  to  send  more 
acknowledgements  to  other  classes. 

Consider  a  class  which  is  UAP  with  degree  b,  the  main  task 
that  eventually  returns  to  the  reference  source  is  defined  to 
belong  to  the  main  chain;  on  the  other  hand,  the  b-1  additional 
flows  which  cause  that  class  to  be  unbalanced  are  perceived  as 
"internal  sources"  (denoted  as  SOURCEU)  which  generate  customers 
to  travel  within  the  network  and  eventually  terminate  at  the 
"internal  sink"  (denoted  as  SINKU) .  It  follows,  as  justified  in 
section  3,  that  all  the  classes  with  UAP  can  be  separated  from 
the  main  chain  to  form  the  UAP  chain  where  the  UAP  chain  is 
defined  as  the  additional  path  through  which  the  "internally 
generated"  customers  (from  SOURCEU)  travel  and  eventually  sink 
(at  SINKU).  For  example,  in  Figure  1-4,  the  classes  (SOURCEU, 
WD1,  WD2,  SINKU)  define  the  UAP  chain.  Note  that  SOURCEU  may 
stand  for  multiple  "internal  sources". 

By  labeling  the  source  and  sink  of  the  main  chain  as  SOURCEM 
and  SINKM,  and  the  source  and  sink  of  the  UAP  chain  as  SOURCEU 
and  SINKU,  one  can  decompose  the  graph  of  a  network  model  with 
UAP  unambiguously  without  referring  to  the  semantics  of  the 
model.  In  other  words,  given  the  labeled  graph  of  an  UAP 
network,  it  is  impossible  to  interchange  one  of  the  UAP  flows 
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2.  MODEL  STRUCTURE 


Without  loss  of  generality,  let's  assume  that  all  customers 
are  homogenous,  i.e.  there  is  a  single  customer  type.  In  Figure 
1-2,  the  single  type  customer  has  a  0.8  probability  of  requesting 
the  read  operation  and  a  0.2  probability  of  requesting  the  write 
operation.  It  would  be  easy  to  relax  this  assumption  to  include 
different  types  of  customers. 

Let  there  be  M  service  facilities  and  C  classes  in  a 
queueing  network.  A  service  facility  may  consist  of  several 
classes  which  allow  customers  to  have  different  sets  of  routing 
probabilities  for  different  visits.  Assume  that  any  sources  and 
sinks  belong  to  class  0.  Let  p,^  denote  the  routing 
probability  which  is  the  fraction  of  the  customers  completing 
service  in  class  i  that  joins  class  j.  i  =  0 ,  . . . ,  C;  j  =  0 , 
...,  C;  and  pQ  0  =  0  by  convention. 

A  main  chain  is  defined  as  the  path  through  which  customers 
travel  according  to  the  defined  routing  probability  and 
eventually  go  out  of  the  system  to  return  to  the  reference 
source.  Since  all  customers  have  been  assumed  to  be  homogeneous, 
there  is  only  one  main  chain  in  the  system.  In  Figure  1-3,  the 
classes  (SOURCEM,  RP1 ,  RD1 ,  WP1 ,  SINKM)  define  the  main  chain. 

A  class  c  customer  of  facility  m  in  the  queueing  network  is 
said  to  be  UAP  with  degree  b  ,i.e.  UAP(c,m)=b,  if  its  output 
splits  into  b  branches  where  b  is  a  real  number  greater  than  one 
but  each  branch  has  a  routing  probability  not  greater  than  one. 


,  - 


■.v. 


approximate  solution  method  was  developed  and  results  of  the 
approximation  were  compared  to  those  of  simulations.  Mean  value 
analysis  approximation  techniques  were  proposed  for  local  area 
distributed  computer  systems  with  UAP  by  Goldberg,  Popek,  and 
Lavenberg  (1983). 

It  is  perhaps  interesting  to  note  at  this  point  that,  quite 
independently  from  the  above  research,  the  authors  of  this  paper 
developed  what  they  have  called  "Distributed  Systems  with 
Unbalanced  Plows"  (Wang  and  Madnick,  1981,  1984)  starting  in 
1981.  The  technique  used  to  model  UAP  is  very  similar  but  a 
different  algorithm  is  used  to  test  the  necessary  and  sufficient 
condition  as  well  as  to  compute  the  closed  network  throughput. 
Moreover,  the  results  for  open  networks  with  UAP,  such  as 
response  time,  have  been  analyzed.  A  syntactic  definition  has 
also  been  given  to  decompose  a  model  uniquely. 

A  terminal-oriented  system  and  a  batch-oriented 
multiprogramming  system  are  modeled  by  Heidelberger  (1982),  and 
local  area  distributed  systems  are  modeled  by  Goldberg  and  others 
(1983)  while  a  hierarchically  decomposed  architecture  is  modeled 
in  the  INFOPLEX  research  (Wang  and  Madnick,  1984).  The 
consistency  reported  from  modeling  these  different  architectures 
provides  further  validation  of  the  modeling  technique.  The  UAP 
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property,  the  convolution  algorithm  can  be  applied  to  solve  the 
nonlinear  equation  iteratively.  Let  O1  denote  the  ith  iteration. 
For  instance,  (EN(Xe))5  denotes  the  throughput  of  EN  at  the  5th 
iteration.  In  the  iterative  procedure,  (X0(U))°  is  estimated 
initially  by  the  lemma  shown  later  and  (Xe(U))1+1  is  determined 
as  follows:  (X0(U))1  +  1  =  (EN(X0))1  +  1  *  V(U)  where  (EN(X0) ) 1  - 1  « 
f ( ( X0 ( U ) ) 1 ) *  This  relationship  is  used  below  to  settle  issue  B. 

B)  The  stability  of  PFMQN  is  unaffected  by  the  presence  of 
closed  chains  (Lazowska,  1984).  Define  the  facility  with  maximum 
open  chain  utilization  (i.e.  the  bottleneck  facility)  as  facility 
I.  It  follows  that  a  closed  network  with  UAP  is  stable  if  and 
only  if  U1(U)<1.  Note  that  U: (U)  *  (X0(U)  /V(U))  *  Dx (U) . 
Denote  v(U)/Dx(U)  as  B,  the  maximum  throughput  of  the.  bottleneck 
facility,  i.e.,  at  saturation.  It  follows  that  a  closed  queueing 
network  with  UAP  is  stable  if  and  only  if  XC(U)  <  B. 

Denote  Dx  (M)  as  the  main  chain  D  value  at  the  bottleneck 

facility  I. The  stability  condition  of  the  closed  network  with  UAP 
can  then  be  identified  with  the  following  four  mutually  exclusive 
and  collectively  exhaustive  cases: 

I)  f (X0 (U)=0 )  *  V(U)  <  B; 

II)  f(Xo(U)-0)  *  V(U)  *  B,  but  Dx (M)  >  0; 

III)  f (X0 (U) =0 )  *  V(U)  >  B,  DX(M)  =  0,  but  f(X0(U)=B)  *  V(U)  <  B; 

IV)  f(Xo(U)«0)  *  V(U)  >  B,  Dx (M)  =  0,  and  f(X0(U)=B)  *  V(U)  *  B. 

Figure  3-3  depicts  the  four  conditions  and  the  lemma  below 

establishes  the  condition  for  stability. 
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Let  a  -  f(Xo(U)«0),  b  •  a*V(U) ,  c  «  f(X0(U)«B),  and  d  »  c*V(U) , 
then  the  four  cases  can  be  rewritten  as  follows: 

I)  b  <  B;  II)  b  >  B,  but  Dx (M)  >  0; 

III)  b  >  B ,  Dx (M)  =  0,  but  d  <  B; 

IV)  b  >  B,  Dx (M)  *  0,  and  d  >  B. 

Lemma:  The  network  is  stable  if  and  only  if  it  is  not  case  IV. 

Proof:  Case  I  states  that  zero  is  given  as  the  initial  estimate 

for  (X0(U))°,  and  (Xo(U))1  «  (EN(Xo))1  *  V(U)  *  b  <  B,  as  shown 
in  Figure  3-3.1.  Since  f  is  CMD  and  a  is  the  upper  bound  of  the 
main  chain  throughput,  it  follows  that  (Xo(U))1  is  bounded 
between  0  and  b  for  all  i.  Therefore,  the  stability  condition 
holds  since  b  <  B. 

Case  II  states  that  zero  is  given  as  the  initial  estimate 

for  (X0(U))°,  but  (Xe  (U) ) 1  2-  B  as  shown  in  Figure  3-3. II.  Since 

there  exists  contention  at  the  bottleneck  facility  I,  i.e.  Dx  (M) 
>  0,  the  actual  throughput,  (Xe (U) )® , will  be  less  than  (Xo(U))1. 
Suppose  a  solution  exists  between  B  and  b,  i.e.  B  <  (X0(U))'°  = 
(EN(X0 )  )*°  *  V(U)  S  b.  It  follows  that  (ENfXo))*  >  B/V(U)  >  0. 
On  the  other  hand,  there  exists  contention  at  facility  I, 
therefore  (EN{X0))’°  =  0  because  the  bottleneck  facility  I  is 
fully  utilized  by  the  open  UAP  chain,  blocking  the  closed  main 
chain  flow  completely.  However,  this  is  contradictory  to  the 
supposition;  therefore,  the  solution  is  bounded  in  the  open 
interval  (0,B)  which  is  less  than  B  and  the  stability  condition 
holds. 
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Case  III  states  that  there  is  no  contention  at  the 
bottleneck  facility,  and  if  B  is  given  as  the  initial  estimate 
for  (X0(U))°,  then  (Xo(U))1  *  d  <  B  as  shown  in  Figure  3-3. III. 
It  follows,  by  CMD,  that  a  solution  exists  in  the  open  interval 
(d,B);  therefore  the  stability  condition  holds. 

Case  IV  states  that  D:(M)  *  0  which  implies  that  the 
bottleneck  facility  I  does  not  contribute  to  the  main  chain 


throughput  at  all.  The  only  impact  it  might  have  is  to  cause  the 
overall  network  to  be  unstable.  If  B  is  given  as  the  initial 
estimate  for  (X0(U))°,  (X0(I J) ) 1  *  d  i  B.  It  follows,  by  CMD,  that 
if  a  solution  exists,  it  must  be  greater  than  or  equal  to  B. 
Thus,  case  IV  violates  the  stability  condition.  Q.E.D. 

Several  points  are  worth  noting: 

a)  The  IS  discipline  was  excluded  since  the  main  chain  and  the 
UAP  chain  do  not  interact  with  each  other  at  the  IS  facility. 
For  networks  with  mixed  disciplines,  the  inflating  factor  for 
the  IS  facility  is  one.  For  networks  with  IS  facilities  only, 
the  UAP  chain  has  no  impact  on  the  main  chain,  therefore,  can 
be  ignored. 

b)  The  Convolution  algorithm,  simple  and  efficient,  is  used  to 
insure  the  stability  condition  as  well  as  to  locate  the 
solution  (Wang  and  Madnick,  1984). 

c)  The  equivalent  closed  network  obtained  from  the  corollary  is 

used  to  calculate  the  "system  response  time"  perceived  by  the 
external  world.  Moreover,  when  the  iterative  procedure  stops, 
G ( 1 ) ,  ...,  G(N)  are  also  available  as  a  byproduct  for 

calculating  useful  performance  measures. 
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d)  It  was  found  (Wang  and  Madnick,  1984)  that  a  bounded 
interpolation  algorithm  takes  an  average  of  2.4  iterations  to 
produce  relative  errors  less  than  0.001  over  10,000 
simulations. 

e)  A  comparative  study  of  the  INFOPLEX  data  storage  hierarchy  has 
been  conducted  to  assess  the  predictability  of  this  technique. 
It  has  been  observed  (Wang  and  Madnick,  1984)  that  the 
analytic  results  are  highly  consistent  with  the  simulations.  A 
closer  examination  of  the  data  shows  that  the  analytic  results 
deviate  from  the  simulations  by  less  than  2%. 
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4.  CONCLUSIONS 


An  analytic  approximation  methodology  has  been  developed  to 
model  distributed  information  systems  with  unbalanced  flows  due 
to  asynchronously  spawned  tasks  (UAP).  The  methodology  allows  us 
to  assess  useful  performance  measures  which  are  crucial  to  the 
success  of  information  systems  utilizing  distributed  processing 
or  local  area  networking.  It  equips  system  designers  with  a  cost 
effective  tool  to  explore  different  design  alternatives. 
Whereas,  it  may  not  be  possible  to  attain  the  steady-state 
results  of  a  single  design  alternative  using  simulation 
economically.  Studies  have  shown  (Wang  and  Madnick,  1984)  that 
this  methodology  produces  the  same  quality  of  results  as 
simulation  with  less  effort  in  addition  to  the  cost 
effectiveness. 

Several  areas  are  open  for  researchers  to  extend  the  work: 
1)  UAP  networks  with  priorities;  2)  software  deadlocks;  and  3) 
hierarchical  decomposition  of  distribution  systems  with  UAP.  The 
payoff  will  be  significant  for  multi-million  dollars  are  usually 
at  stake  in  a  distributed  information  system  project.  Extending 
the  work  to  the  above  areas  will  provide  information  system 
designers  with  an  even  more  powerful  set  of  tools  to  determine 
system  performance  early  in  the  development  process. 
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5.  APPENDIX 


1  Little's  Formula  (22' 


Let  N  be  the  average  over  all  time  of  the  number  of 


customers  in  a  system,  X  be  the  average  arrival  rate  at  the 


system,  and  R  be  the  average  over  all  arrivals  at  the  system  of 


the  system  response  time,  then  N  *  X  *  R.  This  formula  states 


that  the  average  number'  of  customers  in  the  system  is  equal  to 


the  product  of  the  arrival  rate  and  the  average  system  response 


time. 


2  Non-Product-Form  Queueincr  Networks 


If  a  queueing  network  model  does  not  have  a  product  form 


solution,  then  we  usually  must  use  fairly  general  numerical 


techniques,  such  as  solution  of  Markov  balance  equations,  for  its 


solution.  In  this  case  we  shall  find  the  exact  solution  of  the 


network  intractable  unless  it  has  few  service  facilities  and/or 


customers  (20). 


3  Product  Form  Queueing  Networks 


For  the  following  queueing  disciplines,  a  product  form 


solution  exists  for  a  queueing  network:  first  come  first  serve 


(FCFS) ,  processor  sharing  (PS),  infinite  server  (IS),  and  last 
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come  first  serve  preemptive  resumable  (LCFSPR).  If  a  server  has 
a  PS,  IS,  or  LCFSPR  discipline,  then  different  service  time 
distributions  are  allowed  for  different  classes  at  a  service 
facility.  In  this  cases,  the  service  time  distributions  affect 
the  performance  measures  we  shall  consider  only  through  the  mean 
service  time.  If  a  service  facility  has  a  FCFS  discipline,  then 
all  classes  at  the  facility  must  have  the  same  exponential 
service  time  distribution  (1,  16,  17,  20).  A  product  form 
queueing  network  is  one  that  has  a  solution  of  the  following 
form: 


P(S,,...,SM)  =  Pi(SO  ...  Pm(S„)/G(N) 

where  P(S1f  ...  Sm)  is  the  steady-state  probability  of  a  network 
state  in  a  network  with  M  service  facilities,  Pm(S„,)  ,  m  *  1, 
...,  M  is  a  factor  corresponding  to  the  steady-state  probability 
of  the  state  of  service  facility  m  in  isolation.  N  is  the  number 
of  customers  in  the  network,  and  G(N)  is  a  normalization 
constant.  For  an  open  system,  N  can  be  any  number;  for  a  closed 
system,  N  is  a  fixed  number  of  customers  in  the  system.  The 
normalization  constant  G(N)  is  equal  to  the  sum  of  P1(S1)  *  ...  * 
Pm(Sm)  over  all  feasible  network  states. 

4  Single  Chain  Queueing  Networks 

A  single  chain  queueing  network  is  one  with  only  one 
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customer  type.  However,  service  facilities  may  have  several 
classes  which  allow  customers  to  have  different  sets  of  routing 
probabilities  for  different  visits  to  a  service  facility.  Note 
that  although  there  are  several  classes  and  several  routing 
probabilities,  the  only  parameters  in  the  product  form  solutions, 
when  aggregated  to  the  service  facility  level,  are  visit  ratios, 
mean  service  times,  and  number  of  customers  in  the  closed 
queueing  network  case  (20). 

5  Open  Product  form  Single  Chain  Queueing  Networks (OPFSCQN) 

An  OPFSCQN  is  one  with  M  service  facilities  and  C  classes 
and  a  single  chain  that  has  a  product  form  solution.  In 
addition,,  there  are  sources  for  exogenous  arriving  customers  and 
sinks  for  departing  customers.  It  is  assumed  that  customers  from 
exogenous  sources  form  a  poisson  process  with  a  constant  arrival 
rate  X . 

A  remarkable  theorem  by  Jackson  states  that  for  OPFSCQN  with 
a  constant  arrival  rate,  the  network  is  separable  (15),  i.e.  one 
can  compute  a  service  facility's  performance  measures  as  follows 
(20):  Suppose  the  probability  that  an  arrival  customer  enters 

class  c  is  P0 , c  then  it  must  be  true  that 

C 

I  Po.j  *  1 
i-1 

C 

VJ  ■  Po . j  +  z  V,  *  Pl<j  j  ■  1,  •  •  •  ,c 

i«l 


>  .V 


A 
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Suppose  the  system  is  in  the  steady-state,  then  the  system 
arrival  rate  is  equal  to  departure  rate.  Let  X0  denote  system 
throughput,  it  follows  that  X0  ■  X .  Let  X,  be  the  throughput  of 

facility  i,  it  follows  that  X<  «  Xe  *  V,  i  =  1,  ...,  M.  Let  U 

i  ■  X,  *  Si  where  0,  is  the  utilization  of  service  facility  i 

and  Si  is  the  mean  service  time  of  facility  i.  It  is  easy  to  see 
that  an  open  queueing  network  is  stable  iff  Ui  <  1  for  all 
service  facilities  in  the  network.  The  IS  discipline  is  excluded 
from  our  discussion  to  avoid  unnecessary  digression.  The  mean 
queue  length( including  the  one  in  service)  is  N,  *  U(  /  (  1-Ui  ). 

By  Little's  formula,  the  mean  response  time  of  service 

facility  i  is  Ri  ■  N,  /  X<.  It  follows  that  system  response 

time  R  *  R 1  +  ...  +  RM .  The  mean  number  of  customers  in  the 

network  N  *  R  /  XQ.  Note  that  different  formulae  should  be  used 
for  the  IS  discipline.  Thus,  for  OPFSCQN,  one  can  obtain  system 
as  well  as  facility  throughput,  response  time,  and  mean  queue 
length. 

6  Open  Product  Form  Multiple  Chain  Queueing  Networks  (OPFMCQN) 

OPFSCQN  have  a  single  source  and  a  single  sink  and  all 
classes  are  reachable  from  the  source  and  the  sink  is  reachable 
from  all  classes.  It  is  not  necessary,  however,  that  all  classes 
be  reachable  from  one  another.  If  there  are  H  sources  and  the 
classes  are  partitioned  into  H  disjoint  subsets  such  that  for  h  = 

1,  ...,  H,  all  classes  in  subset  h  are  reachable  from  source  h 

* 

and  not  reachable  from  any  other  sources  or  any  other  classes  in 


:.uu 
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any  other  subsets,  then  there  are  H  open  routing  chains  (20).  It 
can  be  shown  that  (20,  31,  33)  that  if  we  have  H  chains,  each 
with  a  poisson  source  with  constant  rate  Xh  ,  h  =  1,  ...,  H,  then 
we  can  treat  the  H  open  chains  as  a  single  aggregate  chain  if  we 
give  that  aggreate  chain  an  arrival  rate  X  *  X,  +  ...  +  XH,  and 
where  class  c  belongs  to  chain  h  in  the  original  network,  make 
the  replacement  P0 . c  * ( X h/X ) *PC . c >  c  *  1,  ...,C. 

7  Closed  Product  Form  Single  Chain  Queueing  Networks  (CPFSCQN) 

A  closed  product  form  single  chain  queueing  network  is  one 
with  M  service  facilities,  C  classes,  and  a  fixed  number  of 
homogenous  customers  that  has  a  product  form  solution.  Several 
algorithms  are  available  for  CPFSCQN,  the  convolution  algorithm 
(4)  remains  the  dominant  algorithm  for  general  purpose  use  (20). 

The  equilibrium  distribution  of  customers  in  CPFSCQN, 
aggregated  at  the  service  facility  level,  is  given  by: 

M  n, 

P(n,,  . . . ,  nm )  =  ( 1/G (N ) )  *  n  (D, ) 

i  =  l 

where  D,  “V,  .  S1f  and  n,  is  the  number  of  customers  of 

facility  i.  It  can  be  shown(5)  that 

P(n,  =  k)  =  (D 1 ) k (G (N- k )  -  Di*G(N-k-l))/G(N) 
where  G(n)  is  defined  as  zero  for  n<0. 

The  mean  queue  length  of  facility  i,  Nir  is  given  by 
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N 

Ni  =  X  (Di ) k  *  G(N-k)  /  G(N) 
k=l 

The  system  throughput,  X0 ,  is  given  by  X0  =  G(N-1)/G(N). 
Therefore,  once  the  values  of  G(i),  ...,  G(N)  are  given,  a  number 
of  useful  performance  measures  can  be  computed. 

8  Convolution  Algorithm 


The  expression  for  G(N)  in  the  equilibrium  distribution 
equation  involves  the  summation  of  C(M+N-1,N)  terms,  each  of 
which  is  a  product  of  M  factors  which  are  themselves  powers  of 
the  basic  quantities.  However,  the  celebrated  Convolution 
algorithm  computes  the  entire  set  of  values  G(l),  ...,  G (N)  using 
a  total  of  N*M  multiplications  and  N*M  additions.  The 
implementation  of  the  algorithm  is  extremely  simple: 

/*  Initialization  */ 

G  ( 0 )  =  1 

for  n  *  1  to  N 

G(n)  =  0 

/*  convolution  */ 
for  m  *  1  to  M 


for  n  *  1  to  N 

G(n )  *  G(n)  +  D(m)*G(n-l) 

/*  end  convolution  */ 
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9  Product  Form  Mixed  Queueing  Networks  (PFMQN) 

Let’s  restrict  a  product  form  mixed  queueing  network  to  be 
one  with  only  one  closed  chain  and  one  open  chain.  Let  "(C)" 
denote  the  closed  chain,  and  "(0)"  denote  the  open  chain.  The 
traffic  intensities  of  facility  i  due  to  the  open  chain  and  the 
closed  chain  are  defined  as 

Pi (0)  =  X0 (0 )  *  V, (0)  *  S, (0) 

Pi (C)  «  XD(C)  *  V, (C)  *  S, (C) 

The  p.g.f.  method  has  been  used  by  Reiser  and  Kobayashi  (31) 
to  provide  important  theoretical  results  for. PFMQN.  It  is  found, 
with  the  p.g.f.  method,  that  1)  The  stability  of  PFMQN  is 
unaffected  by  the  presence  of  closed  chains;  2)  The  open  and  the 
closed  chains  do  not  interact  at  an  IS  service  facility;  and  3) 
The  closed  chain  throughput  is  evaluated  through  a  nonlinear 
function  of  the  open  chain  throughput. 
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